Goods and services are regularly provided by numerous different vendors to numerous different purchasing agencies. Purchasing agencies include a multitude of federal, state and local government agencies spending tax dollars to implement their public functions. Purchasing agencies also include entities in the private sector purchasing the goods and services used to implement their respective business models. Often times a first vendor will sell a first product to a first purchasing agency for a first price, and a second vendor will sell an identical first product to a second purchasing agency at a second albeit higher price. In the past, such sales information has been substantially inaccessible to parties not directly involved in the transaction.
However, if the second purchasing agent knew the price paid for the first item by the first purchasing agent before purchasing the item, then the second purchasing agent could negotiate a lower price, thereby saving money. Similarly, if the first vendor knew that the second purchasing agent had paid more for first product, then the first vendor could take steps to offer future product to the second purchasing agent at lower prices, thereby increasing sales. Furthermore, if the first product was manufactured by a first manufacturer and a second manufacturer produced a second comparable product, and the second manufacture knew which vendors were selling the first product along the corresponding quantities, prices and customers, then the second manufacturer could use that information to develop a marketing and distribution plan for the second product in order to better compete with the first manufacturer, thereby improving the competition between manufacturers.
In the private sector, business transactions between vendors and purchasing agencies are inherently confidential. The private sector is under no obligation to disclose such information. Consequently the availability of information regarding sales transactions between vendors and private sector purchasing agencies is generally unavailable. However, purchases made by government purchasing agencies are not confidential information and may be obtained through the Freedom of Information Act (FOIA) as well as state public records acts. Every month federal, state and local governmental purchasing agencies purchase vast amounts of goods and services. To facilitate these purchases, the government agencies generate an enormous number of purchase orders. Unfortunately, given the disparate nature of the numerous federal, state and local government agencies and the absence of standards in the structure of data affecting purchase orders, the data representing purchases may have significantly variable structures and formats. Furthermore, this data can be retained for a number of years and the structure or format of the data as well as the storage media used to retain the records can vary over time, even within the same governmental purchasing agency. Thus, there is a colossal amount of irregularly formatted data available on the purchase of goods and services by federal, state and local governments through the FOIA, and new data is constantly being generated with the purchase of new goods and services.
Big data is a collection of data sets so large and complex that it becomes difficult to process. While the management of government generated purchase order data may fall within the realm of big data, the availability of the analysis of such data would prove beneficial to the competitive procurement, distribution and manufacture of goods and services.